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writing outcome. (The paper includes 27 lines of the writing protocol 
digcussed and, a ‘chart showing various arerasn⸗⸗ tc error analysis.) 
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Although the diagnostic. — of — is still 


quite primitive, error analysis and protocol analysis hcld promise 


for writing diagnosis. True diagnostic testing does more ‘than aid’in 


placing students; it identifies the nature of needed instruction. — . 
Tests in writing have not keen developed to reflect approaches used 
in: diagnostic. testing in other areas. Reviews of existing 
standardized writing tests suggest shcrtcosings in all three. 
categories of tests identified: objective tests, objective tests” 
corresponding to individualized, writing sequences, and essay tests. 


+ Information about error analysis, 


for writing diagnosis, hasbeen drawn from teachers of English as a 
second language, who have moved from corrective error analysis to 
contrastive analysis and finally to the analysis of errors as clues 
to inner Processes. The other prosising atea, protocol analysis, 
involves analyzing subjects’ cral, descriptions of everything they 


' think’while perforfing-a task, as a means of identifying inner 


Frocesses. and obstacles encountered. An examination of the writing 
protocol of a subject. in an experimental study reveals the process 


". through which a writer understands.a topic. Although erctccols are 
. incomplete representations of inner processes, they*provide far nore 


information about writing processes than does simply examining the 
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The current interest in ‘improving instruction and students! writ- 
ing skills. hes caused teachers to look for some new approaches for- 
diagnosing writing probleme, However, when they have g@ne to the re=. 
search or to other practitioners to determine Pee if anything, : ‘has 
deen done to refine or improve diagnostic: techniques and instruments, 
they have found the field to. be in disarray, For example, in a recent 


survey of forty-four institutions in New York conducted to determine 


what diagnostic. or prescriptive tests were béing used, more than thirty. : 


different: — were identified by the institutions, with seventeen of 


the forty-four citing their own jahouse neste — 


When éducators use the term "diagnosis", the word most frequently 


is — with testing, ueually objective or staiidardized Sectines: 


Some confusion, however, exists about the difference between diagnosis 
and placenent, Most of the commercial tests presently aysilable. 0ane 


claims for their diagnostic potential but closer examination reveals 


that they have only predictive validity; that is, the results of ‘these 


tests may. :be used to make broad generalizations about students! abi- 


lities to write. As a venules students may be placed somewhat accu~ 


. Fately in broad groupings or skill levels, Nevertheless, placing ‘a 


‘student ina curriculum does not necessarily specify the nethods of 


petruetaon that should be. used, with that student nor does it” tdentity. 


_Q Specific problems’ that some pau’ may be having in any one of numerous. 
ihereas of writing. True diagnostic testing, on the other hand,’ should 
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give “information that will identify the nature of the instruction to be 
used and the mprontse items or areas that need attention, Diagnostic 
decisions answer the basic ‘question of what learning activities will . 
best adapt to a learner's individual requirements and thus substantially 
enhance the student's attainment of a chosen goal,. ¢ 


The art of diagnostic testing is not well déveloped * many fields, 


but its lack of sophistication in the field of qriting is most notice- 


able, Psychologists and test measurement experts involved with diag- 


_ nostic testing commonly identify two directions for experimentation.2 


Through this approach attempts are made to discover specific student 
characteristics that interact with methods of instruction in such a 
way that it would be possible to assign different students to different 
pos intone learn the same thing. ‘Although such an ap- 


* proach seems to make good sense, no tests have yet been devised that 


will accomplish this goal successfully, 

The second approaéh in diagnostic testing is to do a —— 
weꝛysis of an ingividual student's performance domain, "3 Here the 
, focus is upon discovering what iia ae prerequisite skills are strong 


or weak and what misinformation or ina opriate associations may in- 


terfere. with ————— activities, Tests, lave been, desiqned . for this 
ee and are conmercially available, ut "only in areas such as math=: 
_ematics, science, and reading, not writing. . . 
A number of — of existing 'Btandardized writing tests have 
been done and all suggest the shortcomings of eXisting tests, ‘One 
of thése reviews, a study’ done for the San Mateo County Community College 
District in California, ‘identified three eetegoxae of tests and found 
substantial limitations in all —— 5 For — standardized ob- 


jective tests such as the College Englich Placement Test (Houghton 
Mifflin 1969); the Purdue Hs School English Test (Houghton Mifflin 


1963) - and the ——eä comprehensive Test of Basic Skills (1970) 


na eS oe — dDuxs/3 
laek diagnostic ‘usefulness for two reasons, First, publishers do not | 
group items testing specific skills so that a score reflecting the 


strengths or weaknesses of a student in a specific skill can be re= 


is, ported, A — flaw in these tests is that too few items on parti 


cular aspects are included to provide reliable’ diagnostic information, 
Only one test, (the Descriptive Tests of Langage Skills DTLS') by 
Educational Testing Service introduced the fall of 1977 seems to 

at leas& partially answer the syidus objections, The DTLS consists 
of five multiple-choice —— Comprehension, Vocabulary, 
Sentence Structure, Logical Relationships, ena Deane.) The tests are 

advertised as being useful for diagnosing students! strengths and — 
weaknesses and do provide more of a break-down, of the individual skills 
~- being Astoa than any of iad other existing tests, Additional tests .. 
may” ‘be developed if these prove useful, ‘ . 

“~~ ina second category, it was found that Objective tests to di- 
agnose specific weaknesses corresponding to textbook or programmed 

* -Andividualized sequences of concepts and writing assignments do have 
diagnostic capabilities; but théae tests are limited to the skills 

in the sequences and tend to be, as do wivoat all writing tents, 

painly focused on. grammatical and mechanical items, | 


A third category, that of essay ests, showed slightly more promise, 


Where attempts have been made elop a curriculum based criteria 
_ for we with such tests, the instruments: do measure: both creative and 
MOERADECOy aspects of writing Once again, “however, the reporting 
methods: for most of these essay tests do riot include means for giving 
specific reports on grammar or mechanics and even thé information on 
as rhetorical strategies is somewhat generalized, 


What all of this reveals, of course, is that the. state of the art 


in ¢ diagnostic testing, as — as writing is’ concerned, is ett quite 
prinitive, We have to realize that the end purpose of diagnostic © 
teating- shoild-be not to screen ‘out prospective students but to match | 

, the available resources of the school to the. strengths and weaknesses. 
Of" the student, Errors in learning are often in.need. of- interpretation 
in light of the particular instructional program in which they gecur. 

In some prograns, a highly proficient level of performance may be, re= 
quired in’ order for a student: to continue, In other progyans, re 
struction is designed so that it is spiral in nature, \When this occurs, 
it is often necessary to have only a minimal level of proficiency in 
order to proceed to new instruction becausé the concept or skill to be « 
_ learned wilh be bauchh auatt and integrated into more advanced learriing. 
‘To some degree,, this spiral theory of instruction could be —— 

to writing; since the-basic rhetorical forme are repeatell at a number ie 
of grade levels. and hence call for on-going reinforcement and review, 

‘Because diagnostic testing, in the traditional Sena, does not at 

the present time offer a — deal of help to the —— instructor, 

we must look for other — for identifying the significance of students! 
strengths and weaknesses, Two areas seem to hold promise for diagnosis 
in writing: error analysis and protocol analysis, 

Information about error analysis has been drawn from those who 
teach English as a second language;6 these people have been engaged for 
some time in a study: of students! speaking and writing errons; and much 
‘of what is used in ESL teaching seems to be ‘applicable to the teaching 
of writing.7 Those involved in ESL teaching, though, have gone through 
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— — attitude in: arriving at their present view oe errors, 
One should. have no difficulty discerning parallels among these attitudes 
and similar ones held by writing teachers, At first, the ESL expert * 


focused on corrective analysis, All érrora were equally, bad and the 


learner was obviously ina a Some teachers even kept lists 


. 
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‘onerged when ‘the student's native language and the second language 


errors was predicted and/or explained by what was called the contrastive — bs. 
‘analysis hypothesis: students will err in the second language when it 


‘collided in a learning situation, The systematic nature of a learner's J 


of recurring errors But little: attempt was made to see any patterns to 
these errors or to seek ‘causes for them ‘ ‘Instead, the errors were 
identified, labeled as bad, and the ‘student warned not to repeat them-- 
an approach Somewhat analogous to the Mbleeding pen" syndrome of some 


composition: teachers, ; a 


ng 
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Then ESL teachers began | to gee that systems existed in the Languages 


and thus teachers became interested in seeing if a pattern of errors a“. bei’. 


' 
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differs fron the: first language. For example, a Vietnamese student 


. might say Ithe writes fas" instead of "he writes fast. " because no con- 


—E clusters. exist’ in Nietnamese, This view corresponds to the 


. theory about dialect interference in native American speakers, which 


in much the same way that reading researchers now view miscues in 


gave @ to the“oufhasis on, pattern drills to make students bi-dialestal, 
The dialect interference view remains a popular explanation of why 
speakers of minority dialects make — errors th writing. However: 


_ the contrastive analysts approach has, lea to: ‘more tolerance of orrorx, 


helping us recognize that many errors may be surface ones and not indi- - # | 
J— J 33. 
cators of a cognitive disability am severe cultural deprivation, 


ESL teachers have now begun to move to a third view of their oy 


. students’ problems in speech and writing; errors are’. looked at 
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‘reading--as clues to: donee — — that lead to —— use, To 

give as dates of hew the first two approaches @iffer from this third. 
——— consider how each would treat the ——— passage: os 


: Marsha, ninety pound lighter, isa ‘Lifetime Weight Watcher 
now, Watching television, she no longer eat -potato chips. 


Although the writer of this passage omits the final "s" on "pound" 


as well as on the verb Neat," let us concentrate on the verb-error, 


The corrective analyst would circle the error in red, explain how 
terrible the: error is to the student and tell the writer not to do 
it again, The contrastive analyst would explain to himself. that the > 
error was the result. of language interference and would dontrane the J 
turo —— to check the hypothesis, ‘The error analyst might see 
the error as an interference problem but that would not be interpreted 
as meaning the writer is locked into the habits. of the first language. 
An the eyes of the error analyst, it is evident that the: student tried | 
to deal with verbs as they are handled in the native ‘dialect; the stu-/ 
dent erred but-at least a strategy was used, ‘If the teacher reacts 
- by looking for several possible causes of the error and then devises 
a strategy for ‘overcoming those influences, the student can learn 
the new strategy and avoid drilling on a single {ten. 
Errors, then, have become more than targets for.-the red pen or - 
the percentile rank, They - be — offer clues to the lin- 
guistic and cognitive processes occurring in students’ minds, One of . 
the primary implications of error analysis pee warch is that since many ah 
errors arise from multiple causes, they are likely to dt nanpaay only if ‘° 


teachers guide students to see all the different routes that led them to 
the error. Mina ———— whose work in error enalysis has helped 


us to become aware of its possibilities, suggests the ——— perspective: 


he 
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- Part of the task of helping ... students master the formal " 

‘verb system therefore depends upon being able to trece the. 

-line of reasoning that has led to erroneous choices rather: 
than unloading upon the student's memory an indifferent bulx 
of tieulties.¢ about verbs, only part of which relates to his | 
difficulties, 


Or, ap. one student wrote after completing a course that had concentrated 
_ exclusively on grammar, ‘We done all the ben itearsons of the verbs for —* 
a semester, put I haven't did any writing yet," The various approaches 


to error analysis are summarized in the chart below, 


Approaches to Learners Errore? 


Issue ; Product Approach Process Approach _ 


Why should one To produce a linguistic - To produce a psycho- . 
study errors? taxonomy of what errors linguistic explanation 
.. learners make — of why a learner make 

an error ; 


N 


What i8 the attitude Errors are "bad," Errors are "good," 
_» toward error? (Interesting only to the (Interesting to the 
— — Aingu*stie theorist)” ~~ ~~ theorist and teacher ‘and a te 
ae . ; useful to thé learner. | 
. as active tests of his a 
: hypotheses) . 


\ 


/ What should we do -——«sOAtttack ‘the’ individual 


Understand the source of 
~ about errors? errors and eliminate errors: tie rule/based 
Th them through drill to - system that produces 
produce overlearning — -.nonestandard forms;, pre 
: : Wide data ny 
; ; _ formation 
What can we hope to * ‘The source of failure: The strategi J 
discover from —— those items on which the —— * oe 
errors? : the learner or the pro- — — r's language, 
. » gram failed is a syst i 
How. can we account for It is principally a Errors — a natural Bare 
the fact that a learner failure to learn the ’ of learning a language; 
makes an error? correct form (perhaps they arise from. learners! 
ey a case of language active strategies: over- 
interference) generalization, ignorance 
vn Om of rule/restrictions, in- 
~ . complete rule. application, 


hypothesizing ‘false concepts 


A teaching perspective: — 
Mat are the emphases A-learning perspective: 
and goals of instruction? eliminate all errors | assist the learner in 
: . by establishing correct, approximating the Target 

- automatic habits; mastery Language, support ‘his - 
of the Target Language 18 active learning strategies’ 
the Boss and) recognize that not all 
_errors will disappear 
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"diagnostic — that of protocol ————— A protocol is a 
description of the - activities, ordered in time, in which a subject 
engages while performing a task, ‘This approach ° polds promise because 
there has been little systpmatic direct observation of fluent writers 
at ‘work and even less of those writers who are not fluent, * Cognitive 
peychologists. developed the technique of protocol analysis as a tool 
for identifying: various psychological processes, Most typically, 
Protocols have been used to identity processes in problem solving. 

But although a protocol isa description, npt every description ofa - 
. task performed is a prototol. For exanple, consider the statement | 
my daughter convinced her ‘father to give — keys." Such a- 
_statement. implies that the. daughter did something but it does not. ‘say 
what or in what order, | Here is.a possible protocol of that situation: 


Father (seated in easy chair, reading evening paper) 


Daughter (enters: room and walke up behind father's chair and . 
' -standa' quietly‘behind it) . 5 


t ee * 


St — 
Father (turns page of newspaper) ; as ; ee 
- . ( — J 
Daaghter (leans over and places arms on ,father's shoulders; 
. “nuzzl@s his left ear) "Dad, are you going out tonight?" 


Father (pauses in reading) No, why?" (raises head slightly) 


Daughter (snuggling closer to father) Well, ‘I have to go. to ‘cheér= 
leading practice and I know'you don't want to have to take me - 2 
and then wait for me to get through." ° : : 


Father ‘(placing paper in dap and trying to see daughter's face) 
_ "Why not?" 


——— ey —— —— on arm of chain) Well, : —— * 
we may have a long practice and you could be watching Monday night ; 
» football--the. Steelers and Cowboys: are playing," 


Father (reaching for TV Guide) "Is that right?" (finds announcement 
in magaziné) "Hmm, you're right; okay, you,can take the car but 
be sure to get in by a1, 


. Daughter (leaning over and ae her father | on cheek) "Thanks, dad, 
a okays, 


“in protocol. 2 alysis, — we: fare interested not — in. what a 


person bays 5 She. ateo in the a as of things. they do tor get to a 


. 


Pein « * 
solution. fw ee s — i ee 


cing “aloud!” ‘protocol aubjecte are “asked: to. ‘si aloud. every, 


thing they think ‘while performing the task, ‘They are asked to say, 


everything: that occurs ‘to them, no matter how trivial it’ may seen. 2 


‘Even with ‘such explicit instructions, subjects Occasionally forget : 


and. ‘Lapes ‘into ailence, completely ‘absorbed . in their task, At such - 


times the analyst has to say, "Remember ,’ tell me everything you are , 
: thinking." After. obtaining: a protoco}, -we analyze it in ai ‘attempt 
* to identify the psychological processes ‘and the obstacles which a a. ae 


subject encountered while: ‘por forn a task, 


The excerpt below pedrides the’ first Awiity Ssoven. ties of 3 a 


‘writing protocol (about "10% of” the: entire protocol); Grown ‘trom the ~~ 


experimental study. of Professors John Re ayes and Linda S. Flower " ) 
at Carnegie=Mellon tniveresty.9 ‘The-writer of. the ‘protocol. was a 


, Volunteer who knew,that she was to. be. involved in a’ writing etudy, 


until arriving for the taping of the session, : NY 
td . 


for about an hour, ‘She also knew that: she would have to "think aloud” 
as she produced a short essay, She did|not know the topic, ‘however, 


(See next page for the protoeoi) 
‘There are several aspects of. this ‘protécol whieh might be pddregned 
ak for purposes of explanation, let us: conipider, on}y. one, ,the. process 


. of understanding a topic. Evidence. that understanding processes are 


at work may be seen as the writing proceeds, For example, lines 1-3 


* show the: writer identifying motivation with grades; then: in lines 6-7, 


she introduces the idea of personal satisfaction but is not certain 


- that this is a source of satisfaction separate from grades, Lines 8-12 ° 


show personal satisfaction being: aneiened a prominent role, but” it is 
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fi _. Writing Protocol 4 : : .| ot 
5 \ — f — ie - ; — | ' 
ie > 5 J . * a x — 


VUV .Ok, um, the issue is motivation and the problem of writing papers. For | ° 
. z = . 


me, motivation here at Carnegie-Mellon is the academic pressure’ and grades.that © 


are Involved, so I'd better put that down, - and grades... Um, they kind‘ of ‘ 
. . ' . oo 


compel me, that's ‘really what ——— um, kind of to impel or start. or o- | 


‘ 
** — wW\ 5 - 


5. ‘By momentum. (Fause. ) Ok, I suppose from the academic pressure of the grades, . 


I's not sure whether, I think personal satisfaction is gmporkents but I'mnot’) .. 


Ce 


, gure “whether that stems from academic ed and grades, or. vnerher⸗ 1 


S8. vould “say personal satisfaction is a — issue. ‘Ok; um. “Oh. . 


» 


L: What are you thinking? 
“Ws I'm trying to think of the first sentence to start with. Um, maybe something 


3 , i 
like, personal satisfaction is the major motivating force in’the writing of my 


_papers- and ‘reports. Ok, I'm trying to think of; ok, i want "to somehow get it fato | 
a: , W 
the academic pressures nou. Um, well maybe not so soon. Ok. Not only do I get | 
f.. 
satisfaction from my grades, but ‘ also eet Satisfaction: in lias in something — 


cd 4 
that is good quality: So, if I'm happy sien I — a good. paper, it really doesn't 


: ‘matter, what kind) of sade I get back on it, if I'm ‘happy with it. So, um, un, 

let's see. Ua, what are the-- I'm’ thinking of, I'm trying to vilate personal: 

sat tefaction —— academic pressure and the grades; but I's not really ire 

* to do it; how to branch ite I'm really ——— a hard time getting started.: 

wets — ria just vette a bunch uF ideas. down, and maybe — to connect them afte 
I finish. Ok.” When I feet that I've ethan a high "quality, * I put in paren- 
theses, peateastonsl, paper, um. to oe ieee I submit it, the peace is not 
‘always necessary dor. the teacher to — the same. Ok, that's kind of 
____3 I'll check with that one. Ok, and-- Let's see what else. Um, but 
{B§dourse, the: reason I'm writing che paper in the first place is for that grade, 


i 
or to relate that back. Those two ideas are very interlocked--maybe that's not 
‘ a 


the right tern. 


> v ; em in Duke/10 


“not” clear how the writer relates it to grades, Lines 13-15 offer | 
tro sources of personatveatisfaction: grades and the production of ° 

high quality work. The two sources of: satisfaction are related in 

lines 15 and. 16 — ‘again in lines 21-23, The basic relationship is 


that if the work ts of good quality, a grade is not necessary for ai) 


tisfaction. A contradictory relationship emerges in lines 24 and 29; 
_ here she notes that grades are the initial motivator for all, essay wri-. 
ting. Then in lines 26-27, these relationships are described as "inter= 
©» locked," | not contradictory, This’ contradiction is realized later in 


rod 


the — and in large measure , the final essay, given below, focuses | 


| on the resolution of ‘this conflict. 


2 


Personal satisfaction is the —— force in the 

writing of my papers and reports. The emphasis on 4,.0's 

at CMU causes grades to become an instinctive motivator for 
myself,  Acquiring-good grades does, at fact, give me per- 

sonal satisfaction, 


The initial motivator in the outset of writing a. paper is 
the fact that a grade will be attached to it upon completion. 
I feel that my role as a student requires all of my efforts 
to be put forth into course work, which includes the writing 
of papers, ~+ 


After I begin writing a paper, the grade emphasis diminishes 
and a higher level of personal satisfaction takes over. When 
I feel that I've written a high-quality or professional paper. 
to be submitted fer grading, it is not mandatory for the tea- 
cher to have the same opinion, But of course, this somewhat 
contradicts my earlier statement that the motivation for writ- 
ing a paper is to achieve the ultimate godl--a good grade, ~ 


Thus, the combination of the grade "initiator" and later a 
higher level of personal satisfaction is what motivates me ° 
to write college papers and reports, 


~ 


v 


In this protocol we glimpse the wwocens of understanding a topic: 
J and the problems involved in that process, Analysis of several pro- | 
tocols in which students perform the same task can give di ener ante 
into how students address a topic and, in turn, we can identify par- 
‘ticular approaches that may help them become more adept in using such 
processes, Therefore, protocol analysis becomes an aid in diagnosing 
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nertornone⸗ rey a wide “avisty of tasks. Typically, 


* are incomplete, Many ‘processes occur during the: perfok ance. ofa task 
* “which the subject can't or doesn't. repért. The beacherts task in ana- 
lyzing the protteol is, of course, “to take the incomplete: report to-, 


— gether, with nt nowledge of the writer's capabilities and. the writing 


: * 
Soe —⸗ — 
: . 
ry 


. process, ! Hikes the underlying Processes or ‘lack of: them by which | 
“ the -ataaont performs the task. The pover of protocol analysie lies . 
ean the richness of, ‘the data, Even though protocols are incomplete, | _ 

_ they provide us. with, fat mgre information about processes by which = 
writing maeke are performed ‘than if we simply examined the outcome, 


— Because truly adaptive and. personalized instruction in writing : 


calle for diagnosis or. student learning difficulties, on a continuing ; 
i _ basis, most of the diagnostic instruments ‘available to us are not ; J 
/aufficient in and of ‘themselves, : Objective tests do little more than 7” 


a 1gok “at error identification; they do not address process and they: do. * 3 ae 
F not provide sufficient discrimination about’ the relative weight of items be 
— to. _nake, the Pesulte useful to the classroom’teacher, As a result,. di- 
— agnosia’ of student’ writing must look to puch promising fields as error * 


analysie and- protocol analysis to yield. important insights which can 


— 


~ be" translated Anto cꝛassroon strategies and —— activities, 
sa 


‘ 3, 
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